Minimizing Image or Video Annotation Effort Using Effective Relevance Models

Humans primarily resort to text or words to describe or present their thoughts when concerning themselves with various forms of multimedia content, such as visual (images or videos) or acoustic (music, etc.). Thus the task of associating text descriptions with images or videos is germane to any human-interpretable discourse around visual multimedia data. Therefore, the multimedia community has focused its attention towards two, in some sense inverse problems, related to the task of associating text with non-text multimedia content (such as images or videos). They are annotation and retrieval. In the former, the goal is to associate text descriptions with the content while in the latter the goal is to retrieve a set of relevant videos or images corresponding to a given text query. This work proposes a novel statistical approach which attempts to address these two questions, also considering the trade-off between the effectiveness of the algorithm and the associated cost of requiring a sizable annotated training data. This effort is directed towards closing the “semantic gap” between representations extracted from images or videos and their corresponding textual descriptions, in an efficient fashion. The traditional image or video annotation or retrieval models rely on the principle of query-likelihood. In such a setting, every document (i.e. the image or video) is associated with a text description. The query-likelihood model assumes the text of the document to have been generated by some underlying probability distribution and the query to have been sampled from that. Such models however, require a sizable amount of training data for a good performance. In this work, we attempt to address this lacunae. We resort to active learning for this. In an active learning setting, using a measure of sample informativeness, we determine an order for labeling the samples in the training data. This order allows us to achieve a good performance even without requiring the entire training set to be labeled. Finally we also propose a novel approach for retrieval of multimedia content, which may be easily extended for annotation as well. Our approach deviates from traditional approaches, since we do not adopt the query-likelihood model for addressing this question. Instead our model, assumes the query and the document to be coming from two different distributions, which are then compared to obtain a measure of relevance. This allows our model more flexibility, in modeling the user’s information needs. The proposed model, is then adapted in the active learning setting described above to achieve efficient and effective retrieval.